Mobile secretary cloud application

ABSTRACT

The invention provides a method, system, and a software program product for assisting a user and/or managing tasks of the user, by a mobile secretary cloud application configured to operate in a mobile client device and cloud server network. The mobile secretary cloud application reads data from another software application and operates at least one of another application and a third application based on the read data. Further, Artificial intelligence is utilized by the mobile secretary cloud application for operating another application and the third application.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present disclosure claims priority to co-pending U.S. patentapplication Ser. No. 16/109,731 filed on Aug. 22, 2018, which claims thebenefit of U.S. Provisional Patent Application 62/590,291 filed Nov. 23,2017, which are incorporated herein by reference in their entireties.

TECHNICAL FIELD

The invention relates to managing tasks of a user through a smartphoneand more particularly to utilizing artificial intelligence in managingtasks of the user.

BACKGROUND

Modern smartphones carry numerous software applications for performingdifferent specialized tasks. For example, a calendar application keepsrecord of activities scheduled for particular dates, an emailingapplication manages emails received and sent by a user, and imagegallery manages images and videos captured using the smartphone.Similarly, there are numerous other applications for managing differentdata relevant to the user.

Many times, it becomes a tedious task for the user to manage suchnumerous applications. For example, the user may need to refer to afirst application and may need to manage a second application based onthe first application. For example, the user may need to access theemailing application for accessing an email comprising details of ameeting scheduled for the user. Based on retrieved details of themeeting, the user may need to revert with an acceptance of the meeting,set an alarm or a reminder for the date of meeting, set a navigationroute for reaching a venue of meeting, or book flight/train/bus forreaching the venue of meeting. Similarly, the user may need to manageseveral activities using different applications installed on hissmartphone, based on information retrieved from one application. Presentstate of art merely comprises management of tasks based on user'sinstructions, either provided manually or via speech.

For example Siri software in the iPhone may take verbal instructionsfrom a user to operate the iPhone.

For example, US20130110519 describes a method to operate an intelligentdigital assistant for assisting a user in task management. Theintelligent digital assistant stores a plurality of predefined domainsindicative of areas of services offered. From a user's request (speechinput), one or more words are derived. Thereafter, a match is identifiedbetween the words derived from the user's request and words associatedwith the plurality of predefined domains. Based on a degree of match, arelevant domain for the user request is identified. Successively, a taskis executed based on steps specified in a task flow associated with therelevant domain and task parameters derived from the user's request.

WO2014197635A2 describes an intelligent automated assistant system. Theintelligent automated assistant system receives a search request from auser. The search request includes at least a speech input and one ormore search criteria for identifying reservable items. In response toreceiving the search request, a plurality of search results,corresponding to reservable items, are presented to the user.Successively, a reservation request is received for one search resultamongst the plurality of search results. In response to the reservationrequest, a reservation procedure is executed to reserve a respectivereservable item.

US20160155442 describes a digital personal assistant configured toperform tasks based on user instructions. Firstly, a user input isreceived, in form of a voice command or a text, for performing a task.Speech recognition is performed on the user input provided as voicecommand. Thereafter, the task is recognized and a registered actionprovider to provide with the task details. Successively, instructionsare sent to the particular action provider for performing the task.Machine learning is used within the process for determining userpreferences, for example if the user selects one action provider 5 timesin a row, on the 6^(th) time that provider is set as the default option.

Thus, the prior art features a number of simplistic workflows andautomations, and even discloses very simple steps for learning thepreference of the user in terms of one criterion. Individual automationsof tasks, however, are not capable of reducing the wholisticadministrative burden of the smartphone user in a significant way.

SUMMARY

It is an object of the invention to address and improve aforementioneddeficiencies in above discussed prior art(s). The object of theinvention is to provide an electronic artificial intelligence softwarebased secretary to every user, at a minimal fraction of the cost that itcosts to hire a human secretary.

The central philosophy of the invention is not to guess what the userwould do, given a certain input into the device. Ordinary consumers arenotoriously inefficient in managing their time. Successful people becomesuccessful partly because they have hired and employ successful personalassistants (PAs) and secretaries. The central philosophy of theinvention is how a professional secretary would handle the input, tomaximize the benefit to the user. In order to achieve this aim,technical software embodiments are produced in the following.

It is an object of the invention to assist a user and/or manageadministrative tasks of the user. Such assistance is provided to theuser by a mobile secretary cloud application configured to operate in amobile client device and cloud server network. Present invention isdescribed using systems, methods, and software program products, asexplained below.

A method for assisting a user and/or managing tasks of the user, by amobile secretary cloud application configured to operate in a mobileclient device and cloud server network, is in accordance with theinvention and characterized by the following steps:

the mobile client device is a smartphone connected to at least one cloudserver network via a wireless communication network,

a data analyzing module of the mobile secretary cloud application readsdata from at least one other application,

a data modifying module of the mobile secretary cloud applicationoperates the at least one other application, based on data read from theat least one other application, and

an artificial intelligence module of the mobile secretary cloudapplication uses artificial intelligence to operate the at least oneother application for assisting a user or to use the at least one otherapplication independently for performing at least one task for the user.

The mobile secretary cloud application may replicate secretaryfunctions. In one case, the at least one other application is an emailor messaging application. The mobile secretary cloud applicationanalyses messages and automatically sorts the messages to differentfolders. Further, the mobile secretary cloud application analysesmessages and selects an indication sound automatically based on contentsof the messages and sender and/or recipients of the messages. In anothercase, the at least one other application is a telephony application, andthe mobile secretary cloud application makes phone calls on behalf ofthe user, automatically answers to messages, and prompts queries onbehalf of the user, using a synthesized voice on a phone line. In yetanother case, the at least one other application is a map application,and the mobile secretary cloud application provides a route calculatorthat provides a combined route of driving and walking based on emails,calendar entries, and location search. In a preferred embodiment thesecretary application uses artificial intelligence to determine whatwould be the optimum way of conducting the three activitiesindividually. Then, further in a preferred embodiment, the secretaryapplication performs a optimum total fit of all of the combinedactivities for the user's day. This may involve for example doing theautomated phone calls during the lunch break of the user, as an eatinguser is less likely to talk. The secretary application will propose adraft email response to a received email, selecting a short matter witha short response for a 10 minute break in the schedule of the user. Incontrast for an entire silent afternoon in the office, the secretaryapplication will present a 30 page project plan for inspection, as thesecretary application knows that now is the convenient time for that.Similarly, for travel, the shortest route that allows the execution ofthe most urgent matters is selected in accordance with the invention.

A method for assisting a user and/or managing tasks of the user, by amobile secretary cloud application configured to operate in a mobileclient device and cloud server network is in accordance with theinvention and characterized by the following steps:

the mobile client device is a smartphone connected to at least one cloudserver network via a wireless communication network,

a data analyzing module of the mobile secretary cloud application isconfigured to read data from at least one other application,

a data modifying module of the mobile secretary cloud application isconfigured to operate a third application, different from the at leastone other application, based on data read from the at least one otherapplication,

an artificial intelligence module of the mobile secretary cloudapplication uses artificial intelligence to operate the thirdapplication for assisting a user or to use the third applicationindependently for performing at least one task for the user.

In one embodiment, the third application is an email or messagingapplication. The mobile secretary cloud application analyses messagesand automatically sorts messages to different folders based on dataderived from a calendar application or an Internet browser. Further, themobile secretary cloud application analyses messages and selects anindication sound automatically based on contents of the messages andsender and/or recipients of the messages, using a text analyzerapplication as the at least one other application.

In another embodiment, the third application is a telephony application,and the mobile secretary cloud application makes phone calls on behalfof the user, automatically answers to messages on behalf of the user,and prompts queries to the user. The mobile secretary cloud applicationperforms such functions using a synthesized voice on the phone line foroutgoing voice and using a voice recognition software application forincoming voice as the at least one other application.

In yet another embodiment, third application is a map application, andthe mobile secretary cloud application provides a route calculator forproviding a combined route of driving and walking based on data from theat least one other application, which data may comprise emails, calendarentries, search application, and Internet browser.

A mobile client device configured to execute a mobile secretary cloudapplication, wherein the mobile client device is a smartphone connectedto at least one cloud server network via a wireless communicationnetwork, and the mobile client device comprises:

-   -   a processor; and

a memory connected to the processor, wherein the processor is configuredto execute the mobile secretary cloud application to:

-   -   read data, using a data analyzing module, from at least one        other application;    -   operate, using a data modifying module, the at least one other        application based on data read from the at least one other        application; and    -   operate, using an artificial intelligence module, the at least        one other application for assisting a user or to use the at        least one other application independently for performing at        least one task for the user.

A mobile client device configured to execute a mobile secretary cloudapplication, wherein the mobile client device is a smartphone connectedto at least one cloud server network via a wireless communicationnetwork, and the mobile client device comprises:

a processor; and

a memory connected to the processor, wherein the processor is configuredto execute the mobile secretary cloud application to:

-   -   read data, using a data analyzing module, from at least one        other application;    -   operate, using a data modifying module, a third application,        different from the at least one other application, based on data        read from the at least one other application; and    -   operate, using an artificial intelligence module, the third        application for assisting a user or to use the third application        independently for performing at least one task for the user.

A software program product stored in a memory medium for assisting auser and/or managing tasks of the user, by a mobile secretary cloudapplication, characterized by the following steps:

-   -   the software program product reads data from at least one other        application;    -   the software program product operates the at least one other        application based on data read from the at least one other        application; and    -   the software program product utilizes artificial intelligence to        operate the at least one other application for assisting the        user or to use the at least one other application independently        for performing at least one task for the user.

A software program product stored in a memory medium for assisting auser and/or managing tasks of the user, by a mobile secretary cloudapplication, characterized by the following steps:

-   -   the software program product reads data from at least one other        application;    -   the software program product operates a third application,        different from the at least one other application, based on data        read from the at least one other application; and    -   the software program product utilizes artificial intelligence to        operate the third application for assisting a user or to use the        third application independently for performing at least one task        for the user.

The best mode of the invention is considered to be a mobile secretaryapplication that is connected to the cloud servers. In the best mode,the cloud network continuously collects and updates self-learning filesfor a number of secretarial tasks, and makes them available to themobile secretary applications on the mobile client terminals. In thebest mode, the mobile secretary application and the cloud network firstdetermine what are the correct draft responses to each incoming taskindividually. Once the individual responses or actions to tasks havebeen determined, the agenda with which these responses or actions aremost efficiently completed with the user in a unit of time, for examplea day or a week, is determined. Once the agenda is determined, thesecretary application and cloud network guide the user through theresponses and activities in accordance with the determined agenda. Somesecretarial activities are also operated on a continuous and instantbasis, for example the playing of indication sounds for arrivingmessages, in the best mode.

The best mode can be applied to a number of uses, some of which mayoperate independently of the cloud network for long periods. Forexample, in the best mode, the selection of an indication sound isautomatically based on contents of the messages and sender and/orrecipients of the messages. In one case, AI is utilized to processcontents of the messages and thereby select an indication sound based oncontents of the messages. For example, the mobile client device of theuser may receive a pleasant message, stating “You did an incrediblejob.” Similarly, the user may receive other pleasant messages, such as“Thank you for your kind support,” “I am glad to hear about yourpromotion,” and “love you my son.” The email and message processingmodule 510, utilizing AI, may learn upon several such messages, and maydetermine that such messages convey pleasant gestures. The email andmessage processing module 510 may learn based on processing of wordspresent in the emails and messages. For example, in present case, thewords “incredible,” “thank you,” “kind,” “glad,” “promotion,” and “love”infer pleasant gestures from a sender, and the email and messageprocessing module 510 may learn such meaning. Based on such learning,the email and message processing module 510 may play uplifting orromantic audio, respectively, as the indication sound.

In contrast, when the mobile client device of the user may receive aharsh message, stating “poor job.” Similarly, the user may receive otherharsh messages, such as “you've passed the deadline,” “I won't toleratesuch careless attitude,” and “hate you.” The email and messageprocessing module 510, utilizing AI, may learn upon several suchmessages, and may determine that such messages convey harsh gestures.The email and message processing module 510 may learn based onprocessing of words present in the emails and messages. For example, inpresent case, the words “poor,” “passed deadline,” “won't tolerate,” and“hate” infer harsh gestures from a sender, and the email and messageprocessing module 510 may learn such meaning. Based on such learning,the email and message processing module 510 may play a loud beep orwarning sound as the indication sound.

Further, the email and message processing module 510 may also derivemeanings based on analysis of smileys/emoticons and special characterspresent in the emails and messages. The special characters may include“! !,” “?,” and other known special characters. For example, the usermay receive a message stating “you again!!” or “what have you done?.”The email and message processing module 510 may learn that harshgestures are implied by such messages. Based on such learning, the emailand message processing module 510 may play the loud beep or warningsound as the indication sound. The indication sound selection embodimentcan be implemented as a cloud server—client device configuration, butalso in the client device only configuration as it requires few or noupdates from the network. In an alternative implementation of the bestmode, the subject line of the email or message or some content of themessage is read out loud to the user, so that the user gets a quickindication of the contents without having to access the message itselfvia the screen.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an embodiment of a flow chart showing a method ofimplementation of a mobile secretary cloud application for operatinganother software application, according to an embodiment.

FIG. 1B illustrates an embodiment of a flow chart showing a method ofimplementation of a mobile secretary cloud application for operating athird software application, using another software application,according to an embodiment.

FIG. 2A illustrates an embodiment of a block diagram of a system forexecuting a mobile secretary cloud application, according to anembodiment.

FIG. 2B illustrates a machine learning embodiment where artificialintelligence is used to distinguish legitimate business emails from spamemails and other emails.

FIG. 3A illustrates an embodiment of a user interface showing a mobilesecretary cloud application operating another software application,according to an embodiment.

FIG. 3B illustrates an embodiment of a user interface showing a mobilesecretary cloud application utilizing another software application foroperating a third software application, according to an embodiment.

FIG. 4A illustrates an embodiment of a flow chart showing a method ofimplementation of a mobile secretary cloud application for operatingseveral other software applications, according to an embodiment.

FIG. 4B illustrates an embodiment of a flow chart showing a method ofimplementation of a mobile secretary cloud application for operatingseveral third software applications, using another software application,according to an embodiment.

FIG. 5 illustrates an embodiment of a block diagram of a systemexecuting a mobile secretary cloud application.

FIG. 6A illustrates an embodiment of a user interface showing a mobilesecretary cloud application operating an email/messaging application.

FIG. 6B illustrates an embodiment of a user interface showing a mobilesecretary cloud application operating an email/messaging application.

FIG. 6C illustrates an embodiment of a user interface showing a mobilesecretary cloud application operating a telephony application.

FIG. 6D illustrates an embodiment of a user interface showing a mobilesecretary cloud application operating a map application.

Some embodiments of the invention are described in the dependent claims.

DETAILED DESCRIPTION

The present disclosure provides a mobile client device, a method, and asoftware program product for assisting a user and/or managing tasks ofthe user. The tasks may be managed by a mobile secretary cloudapplication accessing an application installed in a mobile client deviceof the user. The mobile secretary cloud application may be availableover Google Play™ store for downloading over Android smartphones andover App Store™ for downloading over iOS™ smartphones. The mobilesecretary cloud application may either be available for free or as apaid application.

The mobile secretary cloud application is henceforth explained toutilize Artificial Intelligence for several purposes. Known machinelearning tools/deep learning frameworks may be utilized with or withoutmodifications. A few such known machine learning tools comprise Caffe™,Api.ai™, TensorFlow™, Mahout™, OpenNN™, H20™, MLlib™, NuPIC™ OpenCyc™,Oryx 2™, PredictionIO™, SystemML™, TensorFlow™, and Torch™.

FIG. 1A demonstrates an embodiment 10 of a method of implementation of amobile secretary cloud application for operating another softwareapplication, as a flow diagram. The method could be implemented in asystem identical or similar to embodiment 20 and 50 in FIG. 2 and FIG.5, for example. The end-user of the method could use a user interfaceidentical or similar to that disclosed with embodiment 30, 31, 60, 61,62, and 63 in FIG. 3A, FIG. 3B, FIG. 6A, FIG. 6B, FIG. 6C, and FIG. 6D.

FIG. 2 illustrates an embodiment 20 of a block diagram of a clientsystem 200 executing the mobile secretary cloud application. The systemcomprises interface(s) 202, processor 204, Graphical Processing Unit(GPU) 206, and memory 208. The memory 208 comprises a data analyzingmodule 210, a data modifying module 212, and an Artificial Intelligence(AI) module 214. Different phases of FIG. 1A and FIG. 1B will now beexplained in conjunction with modules of FIG. 2.

Interface(s) 202 are used to interact with or program the system 200.The interface(s) 202 may either be a Command Line Interface (CLI) or aGraphical User Interface (GUI) or both.

The processor 204 may refer to any one or more microprocessors, finitestate machines, computers, microcontrollers, digital signal processors,logic, a logic device, an electronic circuit, an application specificintegrated circuit (ASIC), a field-programmable gate array (FPGA), achip, etc., or any combination thereof, capable of executing computerprograms or a series of commands, instructions, or state transitions.The processor 204 may also be implemented as a processor set comprising,for example, a general purpose microprocessor and a math or graphicsco-processor. The processor 204 may be selected, for example, from theIntel® processors such as the Itanium® microprocessor or the Pentium®processors, Advanced Micro Devices (AMD®) processors such as the Athlon®processor, UltraSPARC® processors, microSPARC™ processors, HP®processors, International Business Machines (IBM®) processors such asthe PowerPC® microprocessor, the MIPS® reduced instruction set computer(RISC) processor of MIPS Technologies, Inc., RISC based computerprocessors of ARM Holdings, Motorola® processors, etc. The control unitdisclosed herein is not limited to employing the processor 204. Thecontrol unit may also employ a controller or a microcontroller and otherelectronics components.

The memory 208 includes a computer readable medium. A computer readablemedium may include volatile and/or non-volatile storage components, suchas optical, magnetic, organic or other memory or disc storage, which maybe integrated in whole or in part with a processor, such as processor204. Alternatively, all or part of the entire computer readable mediummay be remote from processor 204 and coupled to processor 204 byconnection mechanism and/or network cable. In addition to memory 208,there may be additional memories that may be coupled with the processor204 or the GPU (Graphics Processing Unit) 206.

In an embodiment, the system 200 is integrated with a cloud server 216,via a communication network 220. The cloud server may comprise a GPU 218or multiple GPUs. The communication network 220 used for thecommunication in the invention is the wireless or wireline Internet orthe telephony network, which is typically a cellular network such asUMTS—(Universal Mobile Telecommunication System), GSM—(Global System forMobile Telecommunications), GPRS—(General Packet Radio Service),CDMA—(Code Division Multiple Access), 3G-, 4G-, Wi-Fi and/orWCDMA—(Wideband Code Division Multiple Access) network.

In an example, the cloud server 216 may comprise a plurality of servers(not shown). In an example implementation, the cloud server 216 may beany type of a database server, a file server, a web server, anapplication server, etc., configured to store data related to the mobilesecretary cloud application and/or other applications. In anotherexample implementation, the cloud server 216 may comprise a plurality ofdatabases for storing the data files. The databases may be, for example,a structured query language (SQL) database, a NoSQL database such as theMicrosoft® SQL Server, the Oracle® servers, the MySQL® database, etc.The cloud server 216 may be deployed in a cloud environment managed by acloud storage service provider, and the databases may be configured ascloud based databases implemented in the cloud environment.

The cloud server 216 which may include an input-output device usuallycomprises a monitor (display), a keyboard, a mouse and/or touch screen.However, typically there is more than one computer server in use at onetime, so some computers may only incorporate the computer itself, and noscreen and no keyboard. These types of computers are typically stored inserver farms, which are used to realise the cloud network used by thecloud server 216 of the invention. The cloud server 216 can be purchasedas a separate solution from known vendors such as Microsoft and Amazonand HP (Hewlett-Packard). The cloud server 216 typically runs Unix,Microsoft, iOS, Linux or any other known operating system, and comprisestypically a microprocessor, memory, and data storage means, such as SSDflash or Hard drives. To improve the responsiveness of the cloudarchitecture, the data is preferentially stored, either wholly orpartly, on SSD i.e. Flash storage. This component is eitherselected/configured from an existing cloud provider such as Microsoft orAmazon, or the existing cloud network operator such as Microsoft orAmazon is configured to store all data to a Flash based cloud storageoperator, such as Pure Storage, EMC, Nimble storage or the like. UsingFlash as the backbone storage for the cloud server 216 is preferreddespite its high cost due to the reduced latency that is required and/orpreferred for retrieving user data, user preferences, and data relatedto mobile/software applications etc.

The GPU 206 or 218 may refer to an electronic circuit designed tomanipulate and alter computer graphics, images, and memory to acceleratethe analysis and creation of images/patterns. GPUs can be used inembedded systems, mobile phones, personal computers, workstations, gameconsoles, etc. The GPU 206 or 218 may be selected, for example, from AMDGPUs, Nvidia GPUs, Intel GPUs, Intel GMA, Larrabee, Nvidia PureVideo,SoC, etc. In the invention, typically the machine learning or ArtificialIntelligence parts of the processing are configured to be executed bythe GPU, due to the large number of parallel processing or comparativeprocessing required in machine learning.

The system 200 may be configured as a mobile terminal computer,typically a smartphone and/or a tablet that is used to manage tasks ofthe user by operating software applications installed on the smartphone,etc. The system 200 is typically a mobile smartphone, such as iOS,Android or a Windows Phone smartphone. The GPU 206 or is present in thesmartphone or GPU 218 or 518 present in cloud server 216 or 516respectively, may process data of software applications installed on thesmartphone. The processed data may be used for managing the tasks of theuser. All of the three configurations are possible in accordance withthe invention, the first where the GPU 206 is only on the mobile clientdevice 200, the second where the GPU 218 is only on the cloud server216, the third where both the cloud server 218 and the mobile terminaldevice 200 have the GPU 206, 218.

However, it is also possible that the system 200 is a mobile station,mobile phone or a computer, such as a PC-computer, AppleMacintosh-computer, PDA-device (Personal Digital Assistant). The system200 could further be a device having an operating system such as any ofthe following: Microsoft Windows, Windows NT, Windows CE, Windows PocketPC, Windows Mobile, GEOS, Palm OS, Meego, Mac OS, iOS, Linux, BlackBerryOS, Google Android and/or Symbian or any other computer or smart phoneoperating system.

The description encompasses several modules, wherein the modules are tobe interpreted as programmed instructions stored in a segment of thememory 208 or 508, which when executed by the processor 204 or GPU 206,218 performs certain functionalities.

Any features of embodiment 20 may be readily combined or permuted withany of the other embodiments 10, 11, 21, 30, 31, 40, 41, 50, 60, 61, 62,and/or 63 in accordance with the invention.

In phase 102, the data analyzing module 210 of the mobile secretarycloud application or the GPU 206 and/or 218 reads data from anothersoftware application. Another software application indicates a mobileapplication different from the mobile secretary cloud application.Reading the data signifies extracting data entries related to a user.For example, when a calendar application is accessed, all eventsscheduled in a particular month, dates and timings of such events, andpriorities associated with such events are accessed. The events maycomprise meetings, interviews, journeys, birthdays, anniversaries, etc.In one case, the GPU 206 and/or 218 may read the data belonging toanother software application from Random Access Memory (RAM) or cachememory. Further, metadata belonging to the data may also be fetched inone embodiment.

In another embodiment, an email application may be accessed to retrieveall emails present in a mailbox of the user. The data analyzing module210 of the mobile secretary cloud application or the GPU 206 and/or 218may access the email application. All emails present in the mailbox i.e.received, sent, transmission in progress, and recently deleted may beaccessed.

In phase 104, the data modifying module 212 of the mobile secretarycloud application or the GPU 206 and/or 218 operates another softwareapplication based on data read from another software application. Forexample, upon accessing data from a messaging application, the datamodifying module 212 or the GPU 206 and/or 218 may manage messagespresent in the messaging application. In one case, the data modifyingmodule 212 or the GPU 206 and/or 218 may read details related to eachmessage present in the messaging application, such as details ofsenders, content of a message, and contents of other messages receivedfrom a sender. Based on processing of such details, the data modifyingmodule 212 or the GPU 206 and/or 218 may segregate each message into arespective category, such as segregation by category of message,segregation based on sender of messages, and segregation by priority ofmessages.

In another embodiment, upon accessing the emails, the data modifyingmodule 212 or the GPU 206 and/or 218 may manage the emails. The datamodifying module 212 or the GPU 206 and/or 218 may sort the emails basedon priority, time, name of sender(s), name of receiver(s), and contentsof the emails. For example, while the user receives an email towards anacknowledgement of an online order, an instant pop-up message may bedisplayed on the system 200. Successively, based on processing ofcontent present in the email, the data modifying module 212 or the GPU206 and/or 218 may mark the email as read, automatically archive, and/ordelete the email.

In one embodiment, the GPU 206 and/or 218 may utilize ConvolutionalNeural Network (CNN) for processing the data related to the messagingapplication, email application, or other applications. For example,there could be numerous messages present from several senders, and theGPU may perform multi-thread processing for obtaining precise and timelyoutput based on the processing. In one case, the GPU may compriseCombined Unified Device Architecture (CUDA) for enabling themulti-thread processing of the data. For example, the GPU 206, 218and/or processor 204 could generate draft response emails in response toreceived emails. To these emails the GPU 206, 218 and/or processor 204would automatically write text that they have machine learned to berelevant or deduce by Artificial Intelligence that will almost certainlybe present in the response email. For example, if there are multipleemails that address a certain party “Dear Sir/Madam”, the mobilesecretary cloud application will begin the draft response email in thatway too.

In phase 106, the Artificial Intelligence (AI) module 214 of the mobilesecretary cloud application or the GPU 206 and/or 218 utilizesArtificial Intelligence (AI) to operate another software application.The AI module 214 or the GPU 206 and/or 218 operates another softwareapplication for assisting a user and/or to perform a task for the user.In one case, to segregate the messages present in the messagingapplication, the AI module 214 or the GPU 206 and/or 218 utilizes AI forprocessing of different details related to each message. AI may be usedto process content of each message to segregate each message into arespective category. Processing the content of each message may compriseprocessing of title, words, hyperlinks, and dates present in eachmessage. Based on processing of such content, the AI module 214 or theGPU 206 and/or 218 utilizing AI may segregate each message into arelated category.

In addition to analysing the semantic content of messages with AI, themessages can be visually analysed using AI. Consider an example of usingconvolutional neural networks (CNN) for recognizing an email that justarrived in the inbox. CNN refers to artificial neural networks thatmodel visual perception by an animal or a human. The CNN algorithms maybe employed for image recognition tasks preferably emulating the waythat a human secretary views an email in accordance with the invention.The CNN comprises multiple layers of receptive fields that are smallneuron collections configured to process portions of an input image. Theoutput of each layer is successively tiled such that the input regionsoverlap to obtain a representation of the original image. In anembodiment, a deep learning framework called Caffe that uses C++,MATLAB, and Python programming languages is used for implementing theCNN. Caffe is a CNN library that is configured to support both CPU 290and GPU operations. In this example, the GPU 210 or 240 used by themobile secretary cloud application 200 may for example be a NVIDIA GPUwith 15 GB RAM.

A pre-defined dataset comprising, for example, 30000 images oflegitimate business emails and spam emails is used as a training datasetfor training the network. The training dataset comprises labelsassociated with each image. In an example, the training dataset isdownloaded from Kaggle which is a predictive modelling and analyticsplatform. The labelled images 422 and 424 are pre-processed and storedin a Python script format.

In this example, the GPU 206 and/or 218 executes histogram equalizationon the labelled images 422 and 424 of the training dataset. Histogramequalization is a technique used to adjust image intensities by usingthe image's histogram features. Histogram equalization enhances contrastof the images 422 and 424. The images resulting after histogramequalization of the images 422 and 424 are illustrated by images 426 and428, respectively. The GPU 206 and/or 218 performs image resizing toresize the images of the emails, for example, to a 227×227 format. Eachimage of email 430, 432 is labelled after performing the histogramequalization. The training dataset is then divided into 2 subsets. Firstsubset 434 called the training set comprises ⅚^(th) portion of thetraining images that are used for training a model. The second subsetcalled the validation set 436 comprises ⅙^(th) portion of the trainingimages of emails that are used for calculating and validating accuracyof the model. The subsets 434 and 436 are stored in the cloud databaseof the cloud server 216.

Features such as, histogram of oriented gradients (HoG), Scale-invariantfeature transform (SIFT), etc., of the images are extracted from thetraining images 430 and 432 by using a feature extraction software suchas, MATLAB. The extracted image features provide a description offeatures of an object present in an image 430 or 432 that are used inimage classification. Once the subsets 434 and 436 are created, the GPU206/218 generates the mean image for the training data. The GPU 206/218subtracts the mean image from each input image of the training set 434.The GPU 206/218 then performs feature standardization to make eachfeature in the dataset have a zero mean and a standard deviation of 1such that all image data features are normalized. Featurestandardization is used to ensure that measurement comparisons betweenfeatures that may have different units (such as audio signals and pixelvalues of the image data) are normalized. In feature standardization theimage features are centred on a zero mean with a standard deviationof 1. The mean image of the training set 434 is calculated. The meanimage is subtracted from each image in the training set 434. Theresulting value of each image is divided by its standard deviation. Theresulting value of each image feature is hence normalized and can befurther used for creating the training model.

The GPU 206/218 then defines the training model by selecting CNNarchitecture. In this example, the GPU 206/218 uses a CNN architecturemodel such as, Alexnet for defining the model. Alexnet is a CNNframework executed on GPUs implemented in CUDA. CUDA is a parallelcomputing platform and an application programming interface (API) modelcreated by Nvidia that can be used for general purpose processing alongwith graphics processing.

The model is then optimized using a solver algorithm. The solveralgorithm is a type of a stand-alone computer program or a softwarelibrary that is configured to optimize the training model by computingan accuracy of the model using the training dataset. The solveralgorithm computes the accuracy by iteratively using the validation set436. For example, the solver algorithm may use the validation set 436for every 1000 iterations in an optimization process of 40000 iterationsthat takes a snapshot of the trained model at every 5000^(th) iteration.

The GPU 206/218 then performs model training using the results of thesolver algorithm. During the training process, the GPU 210/240 monitorslosses and the model accuracy. In an example, Caffe takes a snapshot ofloss and model accuracy of the trained model at every 5000^(th)iteration. Then the GPU 210/240 plots a learning curve of the losses asa function of the number of iterations as depicted in the graph 438 ofFIG. 2B. Multiple iterations are performed until a steady-state accuracyrate is achieved. For example, as can be seen in the graph 438 thetrained model achieves an accuracy rate of about 90% that stopsimproving after about 3000 iterations.

Once the trained model is ready, the GPU 206/218 starts predictingaccuracy of unseen images from a testing dataset downloaded from aKaggle platform. The GPU 206/218 reads an image of an email from thetesting dataset, processes the image, and calculates a probability ofaccuracy, for example, 0 for non-business email, and 1 for a legitimatebusiness email. For example, if an accuracy rate of 98% is achieved,that image is considered to be of an legitimate business email.

In one embodiment, the model is stored in the cloud database of thecloud server 216 which is accessible by the mobile secretary applicationof the terminal device 200 via the network 220. In another embodiment,the model may be stored in the local memory 208 of the smartphone 200.The GPU 206 and/or 218 analyses an input image of an inbox email 420using the model. The image of the inbox email 420 is compared with thefeatures of a legitimate business email stored in the model. The GPU 206and/or 218 generates an output 245 that identifies the inbox email to bea legitimate business email 245 if the comparison yields a high accuracyrate. In this example, the GPU 206 and/or 218 identifies that the inputimage is of a legitimate business email that is related to a legalmatter. The GPU 206 and/or 218 identifies the business email 245 torelate to a court case the user has. The CPU 204 may compare historicaldata associated with the business email 245 stored in the local memoryof the mobile terminal 200 or the cloud server 216 to determine that theoutput, business email 245 relates to a court case that has the maintrial in 3 days in the calendar application. Hence, the CPU 204 mayrecommend the user to read the email right away. In an embodiment, thebusiness email 245 is placed on the idle screen scale of the mobileterminal 200. A draft response email may be drafted by the mobilesecretary cloud application into the draft folder of the emailapplication, for further editing and approval by the user. Subsequent tothe response, the email 245 is placed into folder “legal” in the emailapplication. The end result of this invention should be the computerizedreplica of the process where a human secretary looks at the email to seewhat it is like, and/or reads all or part of it, and then sorts it tothe relevant folder, and prepares obvious further action, for example adraft response to the arrived email.

By “machine learning” or “artificial intelligence” we mean that thecomputer system has been trained to make determinations based on atraining set of samples, and/or has been tested with a validation set tohave a known error rate as explained before.

Messages or emails can be classified either by analysing themsemantically, or visually as explained before. It is also in accordancewith the invention to use the semantic and the visual AI analysestogether to classify the messages or emails even more accurately. Thevisual AI analysis as explained in the aboce can be applied mutatusmutandis toother secretatial tasks that would be done by the visualperception of the human secretary in accordance with the invention.Preferably the GPU carries out the visual AI tasks, and the CPU carriesout the semantic AI tasks in some embodiments of the invention.

Any features of embodiment 21 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 30, 31, 40, 41, 50, 60, 61, 62,and/or 63 in accordance with the invention.

Although the example discussed herein is provided with reference tousing convolutional neural networks that uses the deep learningframework Caffe, C++, MATLAB, and Python programming languages, theNVIDIA GPU, the Kaggle dataset, and the Alexnet CNN architecture model,it is to be understood that the mobile client 200 and the cloud server216, in another embodiment, may be implemented using any other deeplearning algorithm that uses any other framework, programming language,GPU, dataset, and/or architecture model. The example included herein isdescribed with reference to the publication “A PRACTICAL INTRODUCTION TODEEP LEARNING WITH CAFFE AND PYTHON”, which is included here as areference.

In another embodiment, to segregate the emails present in the emailapplication, the AI module 214 or the GPU 206 and/or 218 utilizes AI forprocessing of different details related to each email. AI may be used toprocess content of each email, and to segregate each email into arespective category. Processing the content of each email may compriseprocessing of title, content, hyperlinks, dates, images, and attachmentspresent in each email. Based on processing of such semantic content, theAI module 214 or the GPU 206 and/or 218 and/or CPU utilizing AI maysegregate each email into a related category, so that emails areorganized into folders of different categories, and email classified tobe in the same semantic category are placed in the same folder.

In one embodiment, the GPU may utilize “Caffe” as the AI for operatinganother software application. In some embodiments, processing of data bythe GPU running the AI may provide leverage for faster processing ofdata. For example, the messages may be simultaneously processed based ontheir title, words, hyperlinks, and dates present therein. The semanticAI analysis of the invention ca be possible implemented in someembodiments with a ready software package such as NLP, Leiki, Than orsimilar semantic AI engine.

It is also in accordance with the invention to segregate emails based onimages taken of the emails only, based on the aforementioned contentanalysis of the emails only, or by using the two methods in a mix.

Any features of embodiment 10 may be readily combined or permuted withany of the other embodiments 11, 20, 21, 30, 31, 40, 41, 50, 60, 62and/or 63 in accordance with the invention.

FIG. 1B demonstrates an embodiment 11 of a method of implementation of amobile secretary cloud application for operating a third softwareapplication, using another software application, as a flow diagram. Themethod could be implemented in a system identical or similar toembodiment 20 in FIG. 2 and embodiment 50 in FIG. 5 for example. Theend-user of the method could use a user interface identical or similarto that disclosed with embodiment 30, 31, 60, 61, 62, and 63 in FIG. 3A,FIG. 3B, FIG. 6A, FIG. 6B, FIG. 6C, and FIG. 6D.

In phase 108, the data analyzing module 210 of the mobile secretarycloud application using the GPU 206 and/or 218 reads data from anothersoftware application. Another software application indicates a mobileapplication different from the mobile secretary cloud application.Reading the data signifies extracting all entries related to the userfrom another software application. For example, while a calendarapplication is accessed, all events scheduled in a particular month,dates and timings of such events, and priorities associated with suchevents are accessed. The events may comprise meetings, interviews,journeys, birthdays, anniversaries, etc. In one case, the GPU 206 and/or218 may read the data belonging to another software application fromRandom Access Memory (RAM) or cache memory. Further, metadata belongingto the data may also be fetched in one case.

In another case, an email application may be accessed to retrieve allemails present in a mailbox of the user. The data analyzing module 210of the mobile secretary cloud application or the GPU 206 and/or 218 mayaccess the mail application. All mails present in the mailbox i.e.received, sent, transmission in progress, and recently deleted may beaccessed.

In phase 110, the data modifying module 212 of the mobile secretarycloud application or the GPU 206 and/or 218 operates a third softwareapplication based on data read from another software application. Thethird software application indicates a mobile application different fromthe mobile secretary cloud application and another software application.In one case, another software application may refer to the messagingapplication and the third software application may refer to the calendarapplication. The data modifying module 212 and/or the GPU 206, 218and/or processor may extract details related to a scheduled meeting,from at least one message stored in the messaging application. Based onextracted details, such as time and place of meeting/event, the datamodifying module 212 or the GPU 206, 218 and/or processor 204 mayschedule a reminder/event in the calendar application. Thereminder/event may be scheduled based on date and time of the meeting,or persons to be contacted or met in the meeting.

In another embodiment, the data modifying module 212 or the GPU 206, 218and/or processor 204 may extract details of a journey planned withfriends. Details of the journey may comprise a date and a flight numberfor booking seats for the journey. Based on extracted details, the datamodifying module 212 or the GPU 206, 218 and/or processor 204 may bookflight tickets using a related application. For example, the extracteddetails may comprise text mentioning “journey begins on Dec. 19, 2017 at9:30 AM from Delhi, India to New York, United States and returning backon Dec. 28, 2017.” Using such date, time, and place of origin anddestination, the data modifying module 212 or the GPU 206, 218 and/oroperator 204 may reserve flight tickets automatically.

In phase 112, the AI module 214 of the mobile secretary cloudapplication or the GPU 206, 218 and/or processor 204 utilizes ArtificialIntelligence (AI) to operate the third software application. The AImodule 214 or the GPU 206, 218 and/or processor 204 operates the thirdsoftware application for assisting a user and/or to perform a task forthe user. In one case, to identify and successively extract detailsrelated to the scheduled meeting, the AI module 214 or the GPU 206, 218and/or processor 204 utilizes AI for processing of different detailspresent in the at least one message. Processing the content of eachmessage may comprise processing of title, words, hyperlinks, dates andtimes, and passwords present in the at least one message. Based onprocessing of such content, the AI module 214 or the GPU 206, 218 and/orprocessor 204 may schedule the reminder/event in the calendarapplication.

In one case, the AI module 214 or the GPU 206, 218 and/or processor 204utilizes Artificial Intelligence (AI) to process the text “journeybegins on Dec. 19, 2017 at 9:30 AM from Delhi, India to New York, UnitedStates and returning back on Dec. 28, 2017.” Using AI, the AI module 214or the GPU 206, 218 and/or processor 204 may identify the date, time,and place of origin and destination details related to the journey. Thedata modifying module 212 or the GPU 206, 218 and/or processor searchesfor a flight matching the journey details. Successively, upon finding asuitable flight, the AI module 214 or the GPU 206, 218 and/or processorbooks a required number of flight tickets.

Choosing which ticket to buy, or present to user for purchase isdetermined similarly as explained before with classifying emails. Ahuman secretary would look at the emails, and see the typing, layout,language and everything to make a visual determination of whether theemail is a legitimate business email. Choosing a ticket is a slightlydifferent process. A human secretary would not look and determine thechoice by appearance. Instead, price, duration, departure time, arrivaltime and transit time in a stopover are the key factors in choosing whatticket to buy.

First a training set of e.g. 30,000 flight tickets is created. Avalidation set comprising ⅙^(th) of the flights is also created. Thenmobile secretary cloud application is taught what flights are acceptableand what are unacceptable. For example, with the validation set, onlythose fights with a low price, short transit time, short duration,coinciding departure and arrival times are selected. This is done forexample by calculating the normalized standard deviations for eachparameter and assigning them weights. The ticket that deviates the leastfrom an acceptable ticket is either bought or presented to the user forpurchase automatically.

Any features of embodiment 11 may be readily combined or permuted withany of the other embodiments 10, 20, 21, 30, 31, 40, 41, 50, 60, 61, 62,and/or 63 in accordance with the invention.

FIG. 3A demonstrates an embodiment 30 of software program product userinterfaces 300 and 306 in accordance with the invention as a screen shotdiagram. The user interfaces show the mobile secretary cloud applicationoperating another software application. The user interfaces could bedisplayed for example on a display screen of a mobile client device i.e.a smartphone.

The software program product is stored on a non-transient memory mediumi.e. the memory 208 of the system i.e. mobile client device 200 and thecloud server 216. The software program product may also be distributedbetween the memory 208 of the mobile client device 200 and the cloudserver 216, so that some parts of the software program product reside inthe memory 208 and some parts of the software program product reside onthe cloud server 216.

In an embodiment, a software program product user interface 300 is shownto comprise a mobile secretary cloud application 302 operating anothersoftware application i.e. a messaging application 304. The mobilesecretary cloud application 302 reads data from the messagingapplication 304 and operates the messaging application 304 based on readdata. The read data may include contents of messages, details of sendersof the messages, date and time of receipt of the messages etc. All suchdetails may be processed using AI and accordingly, the messagingapplication may be operated. In one case, a message may be received froma colleague and the message may ask a user to call his colleague. Duringsuch instance, if it is determined that the user is driving a vehicle,the messaging application may automatically answer to the message thatthe user is driving and that the user shall get in touch after a fewminutes. The user driving the vehicle could be determined based onaccelerometer data of the mobile client device of the user or based on adriving mode set by the user on the mobile client device or fromlocation changes of the mobile client device based on GPS or basestation measurements. Similarly, it is also possible to use images ofemails to determine their type and characteristics as explained before.

In an embodiment, a software program product user interface 306 is shownto comprise a mobile secretary cloud application 308 operating anothersoftware application i.e. a calling application 310. The mobilesecretary cloud application 308 may interact with the callingapplication 310 on user's behalf. In one case, the calling application310 may be connected to an Interactive Voice Response System (IVRS) forbooking flight tickets for the user. The mobile secretary cloudapplication 308 may utilize AI for analyzing all instructions receivedvia the IVRS. For example, the user may be required to set a preferredcommunication language over the IVRS by pressing of a soft key. Themobile secretary cloud application 308 may determine English aspreferred communication language of the user, based on previouscommunication of the user over calls, language of content present inmessage, mails, etc. In one case, the mobile secretary cloud applicationmay analyze user's interaction with the IVRS over several calls, tolearn user's preferences and other user data provided by the user, overthe IVRS. The other user data may comprise personal details, paymentdetails, and details for user identity verification. Successively, themobile secretary cloud application 308 may respond to select English asthe preferred language of the user by selecting the soft key assigned toEnglish. Thereafter, the mobile secretary cloud application 308 mayprovide other inputs to the IVRS, such as providing personal details ofthe user, answering to authentication questions for the user, andselecting a particular department to be contacted via the IVRS either byemulating soft key presses, or reading out answers with synthesizedvoice down the phone line. Thus taking the burden of interacting withthe IVRS, or a human service agent, when the answer to the questionasked by the IVRS or the human service agent is known.

In one preferred embodiment, the mobile secretary application gets ahuman service agent on the line. Remember in the old times, whensecretaries would get people “on the line”, i.e. waiting to thetelephone for the CEO? The mobile secretary cloud application can dothat for the normal consumer. The mobile secretary cloud application 308simply navigates through the choices of the IVRS with selections thatdeviate the least from past selections of the user or users that havebeen recorded. Further, the mobile secretary cloud application mayidentify questions asked by the human service agent, such as “What isyour Date of Birth?” or “What is your Customer number?”, andsubsequently voice recognize these questions, and then dictate the dateof birth of the user, or the customer number of the user down the phoneline with synthesized voice.

Similarly to before, a training set of maybe 30,000 IVRS discussions canbe created. The mobile secretary cloud application and cloud network isthen taught with 5000 IVRS validation set discussions, what is asatisfactory conduct of the IVRS discussion. Typically, this is the IVRSdiscussion that deviates the least from the facts available to themobile secretary cloud application, and the mobile secretary cloudapplication will attempt to conduct that IVRS discussion with a computeror a human service agent on behalf of the user via the phone line to asfar as possible. When the mobile secretary cloud application can nolonger determine a satisfactory response, it may consult the user ordirect that part of the interaction to the human user so that he mayconduct that part of the interaction himself. The mobile secretary cloudapplication may also contain goal based standard call and discussionprotocols, for example “get window seats for flight”. When selected, themobile secretary cloud application calls the airline with the bookingreference obtained from the emails, and conducts the discussion withsoft keys or synthesized voice to obtain a window seat for the user. Ifthat does not work the i mobile secretary cloud application mayautomatically try to obtain the window seat via airline webpages byfilling online forms of the airline or an online travel agent, andoptionally purchasing tickets thereby with the user's credit carddetails stored in the cloud network or the mobile terminal. Of coursethe order in which the call and the website are used could be reversed:the mobile secretary cloud application could try first to obtain thewindow seat via the webpage, but if that does not work it could call theairline as explained before to obtain a window seat. For activities likethis separate training sets and validations sets could be configured,for example by recording 30,000 phone conversations where a window seatis obtained, or by recording 30,000 web sessions where a window seat isobtained.

Any features of embodiment 30 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 31, 40, 41, 50, 60, 61, 62,and/or 63 in accordance with the invention.

FIG. 3B demonstrates an embodiment 31 of a software program product userinterface 312 in accordance with the invention as a screen shot diagram.The user interface 312 shows the mobile secretary cloud application 314utilizing another software application for operating a third softwareapplication. The user interface could be displayed for example on adisplay screen of a mobile client device i.e. a smartphone.

In an embodiment, a software program product user interface 312 is shownto comprise a mobile secretary cloud application 314 utilizing anothersoftware application for operating the third software application, whereanother software application and the third software application isselected from a messaging application 316, a calling application 318,and a map application 320. In one embodiment, the mobile secretary cloudapplication 314 accesses data present in the messaging application 316for operating the calling application 318. For example, the messagingapplication may comprise a message including calling details of an AirConditioning (A.C.) repairing agency. Upon retrieving such details fromthe message, the mobile secretary cloud application 314 mayautomatically call the A.C. repairing agency and may answer to an IVRSsystem over the call, for arranging repair of the A.C. In another case,the messaging application may comprise another message including detailsof a venue for attending a meeting by the user. Based either on a user'srequest or automatically, on the date of the meeting, the mobilesecretary cloud application 314 may operate the map application 320 forsetting a navigation path, to guide the user for reaching the venue ofmeeting. In some embodiments this navigation path is sent to aself-driving car configured to transport the user.

Any features of embodiment 31 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 30, 40, 41, 50, 60, 61, 62,and/or 63 in accordance with the invention.

FIG. 4A illustrates an embodiment 40 of a flow chart showing a method ofimplementation of a mobile secretary cloud application for operatingseveral other software applications. The different phases/stepsmentioned in FIG. 4A are not necessarily performed in mentionedsequence, but could be performed in different sequences orindependently.

FIG. 5 illustrates an embodiment 50 of a block diagram of a system 500executing the mobile secretary cloud application. The system comprisesinterface(s) 502, processor 504, Graphical Processing Unit (GPU) 506,and memory 508. The memory 508 comprises a mail and message processingmodule 510, a call management module 512, and a navigation module 514.In an embodiment, the system 500 is integrated with a cloud server 516,via a communication network 520. The cloud server 516, the GPU 518, andthe communication network 520 are similar to the cloud server 216, GPU218, and the communication network 220 of FIG. 2 respectively. Differentphases of FIG. 4A are now explained in conjunction with modules of FIG.5.

Any features of embodiment 50 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 40, 41, 60, 61, 62,and/or 63 in accordance with the invention.

In phase 402, the mobile secretary cloud application replicatessecretary functions i.e. the mobile secretary cloud application performstasks for the user and/or provides assistance to the user in managingtasks by operating other software application(s) installed on the mobileclient device like a human secretary would.

In phase 404, another software application is an email or messagingapplication 604. The email and message processing module 510 of themobile secretary cloud application 602 or the GPU 506, 518 and/orprocessor 504 analyses messages and emails, and automatically sorts themessages and emails into different folders. Analysis of the messages andemails may be performed by utilizing AI, that processes contents of themessages and emails and other information related to the messages andemails, for sorting the messages and emails into different folders 606a-606 c. The artificial intelligence can be realized by teaching themobile secretary cloud application to distinguish an email from theimage of the email, with training sets and validations sets of images ofemails for example as explained before. Alternatively or additionally,the AI can classify an email based on metadata and semantic content, inwhich case a training set of emails and validations set of emails wouldbe used so, that the artificial intelligence secretary application istaught to classify emails based on their semantic content, and metadata,such as time, recipient, sender and the like. Further these two AImethods can be used together in a mix.

In phase 406, the email and message processing module 510 or the GPU506, 518 and/or processor 504 analyses messages and selects anindication sound (amongst 616 a-616 c) automatically based onclassification, contents of the messages and sender and/or recipients ofthe messages. The email and message processing module 510 or the GPU506, 518 and/or processor 504 may utilize AI to process contents of themessages and may select an indication sound based on the contents. Forexample, the mobile client device of the user may receive a pleasantmessage, stating “You did an incredible job.” Similarly, the user mayreceive other pleasant messages, such as “Thank you for your kindsupport,” “I am glad to hear about your promotion,” and “love you myson.” The email and message processing module 510 or the GPU 506, 518and/or processor 504, utilizing AI, may learn upon analyzingsemantically several such messages, and may determine that such messagesconvey pleasant gestures.

The email and message processing module 510 or the GPU 506, 518 and/orprocessor 504 may be taught to learn or know the semantic meanings ofwords present in the emails and messages. For example, in oneembodiment, the words “incredible,” “thank you,” “kind,” “glad,”“promotion,” and “love” infer pleasant gestures from a sender, and theemail and message processing module 510 or the GPU 506, 518 and/orprocessor 504 may associate a positive meaning to such words. Based onsuch learning, the email and message processing module 510 or the GPU506, 518 and/or processor 504 may play uplifting and positive audio asthe indication sound.

In a contrasting case, the mobile client device of the user may receivea harsh message, stating “poor job.” Similarly, the user may receiveother harsh messages, such as “you've passed the deadline,” “I won'ttolerate such careless attitude,” and “hate you.” The email and messageprocessing module 510 or the GPU 506, 518 and/or processor 504 utilizingAI, may learn upon several such messages, and may determine that suchmessages convey harsh and negative gestures. The email and messageprocessing module 510 or the GPU 506, 518 and/or processor 504 may betaught to attribute a negative meaning to these words present in theemails and messages. For example, in the present embodiment, the words“poor,” “passed deadline,” “won't tolerate,” and “hate” infer harsh andnegative gestures from a sender, and the email and message processingmodule 510 using the GPU 506, 518 and/or processor may be taught to knowor learn such semantic meaning. Based on such learning or knowledge, theemail and message processing module 510 or the GPU 506, 518 and/orprocessor 504 may play a loud beep or warning signal as the indicationsound.

Further, the email and message processing module 510 or the GPU 506, 518and/or processor 504 may also derive meanings based on analysis ofsmileys/emoticons and special characters present in the emails andmessages. The special characters may include “! !,” “?,” and other knownspecial characters. For example, the user may receive a message stating“you again! !” or “what have you done?.” The email and messageprocessing module 510 or the GPU 506, 518 and/or processor 504 may learnand/or distinguish the delivery of a harsh gesture through such message.Based on such learning, the email and message processing module 510 orthe GPU 506, 518 and/or processor 504 may play a warning sound or a loudbeep as the indication sound.

In phase 408, another software application is a telephony application626. Using the telephony application 626, the call management module 512of the mobile secretary cloud application 622 or the GPU 506 and/or 518may make phone calls on behalf of the user, may automatically answer tomessages or compose draft replies, and may answer to incoming calls onbehalf of the user, and may prompt queries and answer to queries onbehalf of the user, using a synthesized voice on a phone line. Forexample, during an IVRS phone call the user's phone number and customernumber may be asked. The call management module 512 or the GPU 506, 518and/or processor 504 may recognize the asked question by voicerecognition software and provide answers to the recognized questionsbased on previous calls or data available in the emails messages, or thefilesystem, and may respond to the IVRS or customer attendant in thecall center by reading the information using synthesized voice.

In another embodiment, the user may need to book flight tickets fromMumbai, India to New York, United States and details of such journey maybe stored in an email or message. Based on approval of the user orautomatically, the mobile secretary cloud application 622 accesses suchdetails for booking the flight tickets for the user. The call managementmodule 512 or the GPU 506 and/or 518 may call a customer care number ofan airline, use a voice synthesizer to read out and voice recognitionsoftware to recognize details for communicating over the call, and thusbook the flight tickets over the phone.

In phase 410, another software application is a map application 634. Thenavigation module 514 of the mobile secretary cloud application 632using the GPU 506, 518 and/or processor 504 using the map application634 provides a route calculator that provides a combined route ofdriving and walking based on emails, calendar entries, and locationsearch. The navigation module 514 or the GPU 506, 518 and/or processor504 upon accessing data present in the emails and the calendar entriesmay determine a date and time of reaching a location by the user. Thenavigation module 514 or the GPU 506, 518 and/or processor 504 maydetermine multiple routes for reaching the location. Further, thenavigation module 514 or the GPU 506, 518 and/or processor 504 mayaccess recent location searches performed by the user, and may utilizesuch information for filtering the multiple routes. In one case, thenavigation module 514 or the GPU 506, 518 and/or processor 504 mayprovide the user with a route which is most familiar to the user, i.e.the user has been through at least once.

The method explained with Graph 438 in FIGS. 2A and 2B and elsewhereearlier in the description can be used with all third applications email510, calls 512, navigation 514. The mobile secretary cloud applicationand cloud server 516 is first provided with a large training set,spanning the diversity of emails, phone conversations, or navigationroutes. The mobile secretary cloud application is then taught with avalidation set of emails, phone conversations or navigation routes whatis an acceptable choice for a user. Once the error rate is diminishedbelow a threshold with a large number of iterations as shown in graph438 of FIG. 2B, the model is deployed to users. Naturally machinelearning files can be continuously maintained, updated and improved onthe cloud server network 516 from which the machine learning files andtheir updates are made available to the terminal devices 200, 500.

Any features of embodiment 40 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 41, 50, 60, 61, 62,and/or 63 in accordance with the invention.

FIG. 4B illustrates a flow chart showing a method of implementation of amobile secretary cloud application for operating several third softwareapplications, using the mobile secretary cloud software application,according to the invention. The different phases/steps mentioned in FIG.4B are not necessarily performed in mentioned sequence, but could beperformed in different sequences or independently. Different phases ofFIG. 4B are now explained in conjunction with modules of FIG. 5.

In phase 412, the mobile secretary cloud application replicatessecretary functions i.e. the mobile secretary cloud application performstasks for the user and/or provides assistance to the user in managingtasks by operating software application(s) installed on the mobileclient device like a professional secretary would.

In phase 414, the third software application operated by the mobilesecretary cloud application 602 is an email or messaging application604. The email and message processing module 510 or the GPU 506, 518and/or processor 504 of the mobile secretary cloud application 602analyses messages and automatically sorts the messages to differentfolders 606 a-606 c shown in FIG. 6A. Along with information present inthe messages and emails, the email and message processing module 510 orthe GPU 506, 518 and/or processor 504 may utilize data of anotherapplication, for example a calendar application, for sorting themessages and emails to different folders 606 a-606 c. In one embodimentthe folders 606 a-606 c have different urgency levels, like one day, oneweek, one month and so on. The mobile secretary cloud application movesthe emails to the different folders 606 a-606 c and in between thefolders 606 a-606 c as the emails become more or less urgent, as newentries are marked into the calendar, and so forth.

In phase 416, the email and message processing module 510 or the GPU506, 518 and/or processor 504 analyses messages and selects anindication sound (amongst 616 a-616 c) automatically based on contentsof the messages and sender and/or recipients of the messages. In onecase, the email and message processing module 510 or the GPU 506, 518and/or processor 504 may utilize AI to process contents of the messagesand may select an indication sound based on the contents, or the folderto which the message is classified.

In phase 418, the third software application is a telephony application626. Using the telephony application 626, the call management module 512or the GPU 506, 518 and/or processor 504 of the mobile secretary cloudapplication 622 makes phone calls on behalf of the user, mayautomatically answer to messages or incoming phone calls, and may answerto queries on behalf of the user, using a synthesized voice on the phoneline. For example, during an IVRS phone call the user's phone number andcustomer number may be asked. The call management module 512 mayrecognize the question by voice recognition software and find requestedinformation in a digital wallet of the user or other application storingsuch detail, and may respond on the IVRS by reading the informationusing the synthesized voice. In another embodiment, the user may need tobook flight tickets from Mumbai to New York and details of such journeymay be stored in an email or message. Based on approval of the user, thecall management module 512 or the GPU 506, 518 and/or processor 504access such details for booking the flight tickets for the user. Thecall management module 512 or the GPU 506, 518 and/or processor 504 forexample calls a customer care number of an airline, uses a voicesynthesizer to read such details for communicating over the call, andthus books the flight tickets. Alternatively, the mobile secretary cloudapplication may be configured to operate an e-commerce application tobook the flights. The acceptable flight or acceptable flights forpurchase can be determined as explained before, by providing a trainingset and validation set of flights and teaching the mobile secretarycloud application to select an acceptable flight that, for exampledeviates from the model acceptable flight the least in a number ofparameters such as duration, price, departure and arrival time and/orstopover duration.

In phase 420, the third software application is a map application 634.The navigation module 514 of the mobile secretary cloud application orthe GPU 506, 518 and/or processor 504 using the map application providesa route calculator that provides a combined route of driving and walkingbased on emails, calendar entries, and location search performed by theuser for conducting the activities of the day at various locations andtimes. The navigation module 514 or the GPU 506, 518 and/or processor504 upon accessing data present in the emails 636 and the calendarentries 638 may determine a date and time of reaching a location by theuser. The mobile secretary cloud application may determine multipleroutes for reaching the location. Further, the navigation module 514 orthe GPU 506, 518 and/or processor 504 may access recent locationsearches 640 performed by the user, and may utilize such information forfiltering the multiple routes. In one case, the navigation module 514 orthe GPU 506, 518 and/or processor may provide the user with a routewhich is most familiar to the user, or which route when followed allowsthe user to complete the maximum number of errands or tasks on his to dolist.

The aforementioned secretarial process of optimizing routes, agendasand/or timetables naturally results in the knowledge of events that theuser has actually visited. The location and time of the mobile stationcan be followed and calendar entries and Internet search can beharnessed along with call application data to construct a history ofevents that the user has attended. Many of these events are likely to berecurring: conferences happen every year, the barber has to be visitedonce per month, the dentist once per year, the accountant needs to becontacted every 3 months to prepare a quarterly report and so forth. Themobile secretary cloud application can automate the rebooking of theuser into these events. In the first stage, the mobile secretary cloudapplication may populate the calendar with entries when the event takesplace or should take place. The mobile secretary cloud application maysearch the Internet to determine the 2019 dates of a conference that theuser attended in 2018, and mark them up in the calendar. The mobilesecretary cloud application may also register the user to thisconference automatically or upon approval of the user. The mobilesecretary cloud application may book barber and dentist appointments ata convenient time and place automatically or upon approval of the useras explained before via phone or the webpage of the barber or dentist.All of these activities can be done without or almost without any userparticipation by the mobile secretary cloud application merely repeatingrecorded telephone conversations or webpage sessions and adapting themwith AI to the current or upcoming situation. If plans change the mobilesecretary cloud application may automatically cancel activities orevents on behalf of the user, by similarly executing web sessions orphone conversations of cancellation on behalf of the user, adapted withAI to the current situation.

Any features of embodiment 41 may be readily combined or permuted withany of the other embodiments 10, 20, 21, 30, 31, 40, 50, 60, 61, 62,and/or 63 in accordance with the invention.

FIG. 6A demonstrates an embodiment 60 of a software program product userinterface 600 in accordance with the invention as a screen shot diagram.The user interface 600 shows the mobile secretary cloud application 602operating an email/messaging application 604. The user interface 600could be displayed for example on a display screen of the mobile clientdevice i.e. a smartphone 500.

In an embodiment, the software program product user interface 600 isshown to comprise the mobile secretary cloud application 602 operatingthe email/messaging application 604. The mobile secretary cloudapplication 602 reads data from the email/messaging application 604 andoperates the email/messaging application 604 based on read data. Theread data may include contents of messages, details of senders of themessages, date and time of receipt of the messages etc. All such detailsmay be processed using AI and accordingly, the messaging application maybe operated. In one case, based on processing of the read data, themobile secretary cloud application 602 may store official work relatedmails and messages into folder 606 a, emails and messages from friendsand families into folder 606 b, and other emails and messages, such asfrom banks, telecom operators, and other service providers into folder606 c. For example, the messages and emails may be sorted into differentfolders based on the senders, month of receipt, and details of themessages/emails. The mobile secretary cloud application may also takeimages or photographs of the emails to graphically process the emails asexplained before in association with FIGS. 2A and 2B. Similarly, thephotographic/visual classification of emails or messages may be donetogether with the semantic classification so that these two methods areused in a mix. It is also in accordance with the invention for themobile cloud secretary application to search the Internet to gainfurther information about sender, recipient, or the like to assist inthe message and/or email classification.

Any features of embodiment 60 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 40, 41, 50, 61, 62,and/or 63 in accordance with the invention.

FIG. 6B demonstrates an embodiment 61 of a software program product userinterface 610 in accordance with the invention as a screen shot diagram.The user interface 610 shows the mobile secretary cloud application 612operating an email/messaging application 614. The user interface 610could be displayed for example on the display screen of the mobileclient device i.e. a smartphone.

In an embodiment, the software program product user interface 610 isshown to comprise the mobile secretary cloud application 612 operatingthe email/messaging application 614. The mobile secretary cloudapplication 612 analyses messages and selects an indication soundautomatically based on contents of the messages and sender and/orrecipients of the messages. In one case, the email and messageprocessing module 510 or the GPU 506, 518 and/or processor 504 utilizesAI to process contents of the messages and selects an indication soundbased on the contents. For example, the mobile client device of the usermay receive a pleasant message, stating “You did an incredible job.”Similarly, the user may receive other pleasant messages, such as “Thankyou for your kind support,” “I am glad to hear about your promotion,”and “love you my son.” The email and message processing module 510 orthe GPU 506, 518 and/or processor 504, utilizing AI, may be taught thatthe semantic meaning of such messages convey pleasant gestures. Theemail and message processing module 510 or the GPU 506, 518 and/orprocessor 504 may also continuously learn and update machine learningfiles, or deep learning files on the terminal or in the cloud, based onprocessing of words present in the emails and messages. Typically insome embodiments these machine learning files, or deep learning files orArtificial Intelligence files contain training set and/or validation setdata. For example, in present embodiment, the words “incredible,” “thankyou,” “kind,” “glad,” “promotion,” and “love” infer pleasant gesturesfrom a sender, and the email and message processing module 510 or theGPU 506, 518 and/or processor may learn such meaning, and/or attributesuch semantic meaning to these words. Based on such learned orattributed meaning, the email and message processing module 510 or theGPU 506, 518 and/or processor 504 may play a positive and upliftingaudio 616 a with a loudspeaker as the indication sound.

It is in accordance with the invention to have the Artificialintelligence files used to train and validate the ArtificialIntelligence process on the terminal, in the cloud and/or distributedbetween both of them. Artificial intelligence files, deep learningfiles, machine learning files and self-learning files are usedinterchangeably, and refer to the files used to establish the trainingand validation process producing the artificial intelligence asexplained before for example with FIG. 2B.

In a contrasting case, the mobile client device of the user may receivea harsh message, stating “poor job.” Similarly, the user may receiveother harsh messages, such as “you've passed the deadline,” “I won'ttolerate such careless attitude,” and “hate you.” The email and messageprocessing module 510 or the GPU 506 and/or 518, utilizing AI, may learnupon several such messages, and may determine that such messages conveydissatisfaction from the sender. The email and message processing module510 or the GPU 506, 518 and/or processor 504 may also continuously learnbased on processing of words present in the emails and messages, andupdate the machine learning files on the terminal device and/or thecloud server. For example, in present case, the words “poor,” “passeddeadline,” “won't tolerate,” and “hate” infer dissatisfaction from asender, and the email and message processing module 510 or the GPU 506,518 and/or processor may know, or learn such meaning. Based on suchlearning or knowing the semantic meaning, the email and messageprocessing module 510 using the GPU 506, 518 and/or processor 504 mayplay a loud beep 616 b or warning sound as the indication sound with aloudspeaker.

Further, the email and message processing module 510 or the GPU 506, 518and/or processor 504 may also derive meanings based on analysis ofsmileys/emoticons and special characters present in the emails andmessages. The special characters may include

“! !,” “?,” and other known special characters or emoticons. Forexample, the user may receive a message stating “you again! !” or “whathave you done?.” The email and message processing module 510 or the GPU506, 518 and/or processor may learn delivery of harsh gesture throughsuch message, or a positive gesture if the emoticon has a positivesemantic meaning. Based on such knowing or learning, the email andmessage processing module 510 or the GPU 506, 518 and/or processor 504may play a positive uplifting signal for positive emoticons and anegative audio signal such as a loud beep 616 b as the indication soundfor negative emoticons. Similarly, other indication sounds 616 c may beused for different special characters or emoticons indicating otherexpressions and/or gestures of the emails and messages received by theuser.

Any features of embodiment 61 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 40, 41, 50, 60, 62,and/or 63 in accordance with the invention.

FIG. 6C demonstrates an embodiment 62 of a software program product userinterface 620 in accordance with the invention as a screen shot diagram.The user interface 620 shows the mobile secretary cloud application 622operating the calling application 626. The user interface 620 could bedisplayed for example on the display screen of the mobile client devicei.e. a smartphone.

In an embodiment, the software program product user interface 620 isshown to comprise the mobile secretary cloud application 622 operating acalling application 626. The mobile secretary cloud application 622 mayinteract with the calling application 626 on user's behalf. In one case,the calling application 626 may be connected to an Interactive VoiceResponse System (IVRS) for bookings flight tickets for the user. Themobile secretary cloud application 622 may utilize AI for analyzing allinstructions or voice prompts received via the phone line from the IVRS.For example, the user may be required to set a preferred communicationlanguage over the IVRS by pressing of a soft key or number key. Themobile secretary cloud application 622 may determine English aspreferred communication language of the user, based on user data anduser preferences 624 of the user. Successively, the mobile secretarycloud application 622 may respond to select English as preferredlanguage of the user. Thereafter, the mobile secretary cloud application622 may provide other inputs to the IVRS, such as providing personaldetails of the user, answering to authentication questions for the user,and selecting a particular department to be contacted via the IVRS. Themobile secretary cloud application 622 may retrieve all requiredinformation from the user data and user preferences 624 of the user. Forexample, the user data may comprise travelling details, personal detailsand payment details of the user. Thus, the mobile secretary cloudapplication 622 may provide the travelling details over the IVRS and mayfinally provide the payment details, over the IVRS, for booking a flightticket for the user.

Any features of embodiment 62 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 40, 41, 50, 60, 61,and/or 63 in accordance with the invention.

FIG. 6D demonstrates an embodiment 63 of a software program product userinterface 630 in accordance with the invention as a screen shot diagram.The user interface 630 shows the mobile secretary cloud application 632operating the map application 634. The user interface 630 could bedisplayed for example on the display screen of the mobile client devicei.e. a smartphone.

In an embodiment, the software program product user interface 630 isshown to comprise the mobile secretary cloud application 632 operatingthe map application 634. The mobile secretary cloud application 632using the map application 634 provides a route calculator that providesa combined route of driving and walking based on emails, calendarentries, and location search. The emails may be accessed from theemail/messaging application 636, calendar entries may be accessed fromthe calendar application 638, and location search performed by the usermay be accessed from a location search application 640 or Internetbrowser. The mobile secretary cloud application 632 upon accessing datapresent in the emails and the calendar entries may determine a date andtime of reaching a location by the user. The mobile secretary cloudapplication 632 may determine multiple routes for reaching the location.Further, mobile secretary cloud application 632 may access recentlocation searches performed by the user, and may utilize suchinformation for filtering the multiple routes. In one case, the mobilesecretary cloud application 632 may provide the user with a route whichis most familiar to the user, i.e. the user has been through at leastonce.

In one case, the mobile secretary cloud application 632 may identifymultiple calendar entries present in the calendar application 638, for aparticular day. The user may need to visit multiple locations forattending different meetings scheduled on the day. In such case, themobile secretary cloud application 632, using the map application 634may provide a route map to the user, such that a least distance or timewill need to be travelled to visit all the locations.

Any features of embodiment 63 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 40, 41, 50, 60, 61and/or 62 in accordance with the invention.

In the preceding description it has been described how the mobilesecretary cloud application arrives at good or optimum outcomes insecretarial tasks using machine learning and artificial intelligence. Asexplained before, this is typically achieved by taking a training set ofsubstantial size, typically 30,000 samples or so, and teaching thecomputer with a validation set of about 5000 samples what is anacceptable choice and what is not an acceptable choice. This principleis applied to different secretarial tasks regardless of the data formatof the output. For example, if the secretarial task is looking at emailsand judging which emails are genuine and important, the sample sets usedwill be semantic data. i.e. words and content of the email, its metadata(sender, recipient, time, title) and also optionally the image of theemail. The type setting, colors, layout, all visual factors contributeto the decision of how to characterize the email.

If the secretarial task is interaction on the phone, the training setsand validation sets will likely be audio soundtrack and dial toneentries. If the secretarial task is route planning, the training andvalidation sets will be the routes. If the secretarial task is bookingflight tickets, the training and validations sets will be flighttickets.

Typically, the mobile secretary cloud application 602, 612, 622, 632,will be conducting Internet searches from time to time automatically tofind a result(s) that would be satisfactory to the AI model in theterminal device and cloud network. This means that if a good enoughroute, flights, or the like is not found, the secretary application cancontinue to conduct Internet searches to find a satisfactory result forthe user on his behalf.

In the aforementioned description, the mobile secretary cloudapplication 602, 612, 622, 632 has been described, and how the mobilesecretary cloud application arrives at good or optimum outcomes insecretarial tasks using machine learning and artificial intelligence. Itis also in accordance with the invention to combine optimized results ofindividual secretarial outcomes to arrive at an optimized agenda for anentire day for the user. This is achieved as follows. The terminalcomputer 200, 500 and cloud network 216, 516 is presented with atraining set of 30,000 or so daily agendas, and a validation set of afew thousand agendas to teach the mobile secretary cloud application602, 612, 622, 632 and the cloud server network 216, 516 to distinguishbetween a good daily agenda for the user, and an unacceptable agenda forthe user. When the mobile secretary cloud application determines theindividual outcomes to the secretarial tasks, it will then fit thesetasks to a daily agenda. This daily agenda will then be compared withthe AI model, and the agenda that is accepted by the trained AI model ispresented to the user with the secretarial outcomes in accordance withthe agenda. In some embodiments an agenda for a week, month and/or year,or any unit of time is similarly composed, using an AI model asexplained before mutatis mutandis.

For each secretarial task the mobile secretary cloud application can betrained with deep-learning datafiles. These deep learning data files mayreside on a cloud server to which the client device, i.e. typically amobile phone is connected to via Wi-Fi or cellular connection. Themobile secretary cloud application can be trained to conduct newsecretarial tasks on behalf of the user by providing new sets of deeplearning data files. Similarly secretarial activities that the mobilesecretary cloud application already performs can be improved by updatingdeep learning data files. The training and validation datasets for anyactivity can be updated from time to time so that the secretarialapplication recognizes the need for an appropriate secretarial task withthe best accuracy, and that the secretarial application fulfills thetask with the best accuracy.

The mobile secretary cloud application may record tasks that it performseither in the form of a computing session, a recording of a synthesizedphone conversation, and/or text, and/or images of messages or sessionsthat the mobile secretary cloud application performs. The recordings ofpast tasks may be used to enrich the deep learning data files. The deeplearning data files may reside on either the client device such as themobile phone or in the cloud in accordance with the invention. The deeplearning data files may be shared by one or more cloud servers betweendifferent users in some embodiments. In some embodiments the origin ofthe deep learning data files used to train the mobile secretary cloudapplication is concealed from the user, in order to protect the privacyof those users from whom the deep learning data files originate. It isalso in accordance with the invention that every user generates all orsome of their own deep learning data files.

The invention has been explained in the aforementioned and sizeableadvantages of the invention have been demonstrated. The invention allowsthe mobile secretary cloud application to replicate the secretarialfunctions to a normal consumer at a very small cost. The mobilesecretary cloud application produces draft responses to emails andmessages, interacts on the phone on behalf of the user, and makespurchases such as flight tickets on behalf of the user. The inventivesecretarial application also sorts and archives emails and messages todifferent folders, arranges tasks for the user based on priority andurgency, and selects routes and agendas for the day. Similarly, thesecretary application 602, 612, 622, 632 updates data from oneapplication to the other, for example emailed agreements of meetings tothe calendar and provides different indication sounds alerting the userto the category or vibe of an incoming message. Any secretarial task canbe performed by the intelligent mobile secretary application 602, 612,622, 632 and cloud network 216, 516 as long as the system is trained toproduce and distinguish an acceptable secretarial work product withtraining and validations sets of the said secretarial work product inaccordance with the invention.

The invention has been explained above with reference to theaforementioned embodiments. However, it is clear that the invention innot only restricted to these embodiments, but comprises all possibleembodiments within the spirit and scope of the inventive thought and thefollowing patent claims.

REFERENCES

-   US 2013/0110519 A1 DETERMINING USER INTENT BASED ON ONTOLOGIES OF    DOMAINS, published on May 2, 2013, Adam et al.-   WO 2014/197635 A2 INTELLIGENT AUTOMATED ASSISTANT, published on Dec.    11, 2014, Thomas et al.-   US 2016/0155442 A1 EXTENDING DIGITAL PERSONAL ASSISTANT ACTION    PROVIDERS, published on Jun. 2, 2016, Vishvac et al.-   A PRACTICAL INTRODUCTION TO DEEP LEARNING WITH CAFFE AND PYTHON,    published on Jun. 26, 2016, Adil Moujahid.

1. A non-transitory computer readable memory including instructions forperforming operations for a software program product to operate with amobile client device connected to a cloud server network via a wirelesscommunication network, the operations comprising: reading, by thesoftware program product, data from a messaging application; operating,by the software program product, the messaging application, based ondata read from the messaging application, wherein operating themessaging application includes analyzing and sorting messages; andusing, by the software program product, machine learned data to operatethe messaging application for assisting a user or to use the messagingapplication independently of the user for performing tasks on behalf ofthe user, wherein the tasks include all of: answering an inbound phonecall; answering queries included in inbound messages automatically viathe messaging application; and organizing an efficient agenda for theuser using the machine learned data.
 2. The non-transitory computerreadable memory of claim 1, wherein the software program productanalyses messages from the messaging application and selects anindication sound automatically based on one or more of: contents of themessages and sender of the messages or recipients of the messages. 3.The non-transitory computer readable memory of claim 1, wherein themessaging application includes a map application and wherein thesoftware program product provides a route calculator that provides acombined route of driving and walking based on e-mails, calendarentries, and location search performed by the user.
 4. A non-transitorycomputer readable memory including instructions to perform a method forassisting a user or managing tasks of the user, by software programproduct configured to operate with a mobile client device connected to acloud server network via a wireless communication network, wherein themethod comprises: reading, by the software program product, data from anemail or calendar application; operating, by the software programproduct, a messaging application, different from the email or calendarapplication, based on the data read from the email or calendarapplication, wherein operating the messaging application includesanalyzing and sorting messages; using, by the software program product,machine learned data to operate the messaging application for assistinga user or to use the messaging application independently of the user forperforming tasks on behalf of the user, wherein the tasks include allof: answering an inbound phone call; answering queries included ininbound messages automatically via the messaging application; andorganizing an efficient agenda for the user using the machine learneddata.
 5. The non-transitory computer readable memory of claim 4, whereinthe software program product analyses messages from the messagingapplication and selects an indication sound automatically based on atleast one of contents of the messages and senders of the messages orrecipients of the messages, using a text analyser application.
 6. Thenon-transitory computer readable memory of claim 4, wherein the softwareprogram product provides a route calculator via a map application thatprovides a combined route of driving and walking based on data from theemail or calendar application comprising e-mails, calendar entries, asearch application, and an Internet browser.
 7. A mobile client deviceconfigured to execute a software program product, wherein the mobileclient device is a smartphone connected to at least one cloud servernetwork via a wireless communication network and the mobile clientdevice comprises: a processor; and a memory connected to the processorand storing instructions for the software program product, wherein theprocessor is configured to execute the software program product to: readdata, from a messaging application; operate, the messaging applicationbased on data read from the messaging application, wherein operating themessaging application includes analyzing and sorting messages; andoperate, the messaging application with machine learned data forassisting a user or to use the messaging application independently ofthe user for performing tasks on behalf of the user, wherein the tasksinclude all of: answering an inbound phone call; answering queriesincluded in inbound messages automatically via the messagingapplication; and organizing an efficient agenda for the user using themachine learned data.
 8. The mobile client device of claim 7, whereinthe software program product analyses messages and selects an indicationsound automatically based on one or more of contents of the messages,senders of the messages, and recipients of the messages.
 9. The mobileclient device of claim 7, wherein the software program product providesa route calculator via a map application that provides a combined routeof driving and walking based on e-mails, calendar entries, and locationsearch performed by the user.
 10. A mobile client device configured toexecute a software program product, wherein the mobile client device isa smartphone connected to at least one cloud server network via awireless communication network, and the mobile client device comprises:a processor and a memory connected to the processor and storinginstructions for the software program product, wherein the processor isconfigured to execute the software program product to: read data, froman email or calendar application; operate, a messaging application,different from the email or calendar application, based on the data readfrom the email or calendar application, wherein operating the messagingapplication includes analyzing and sorting messages; operate, themessaging application with machine learned data for assisting a user orto use the messaging application independently of the user forperforming tasks on behalf of the user, wherein the tasks include allof: answering an inbound phone call; answering queries included ininbound messages automatically via the messaging application; andorganizing an efficient agenda for the user using the machine learneddata.
 11. The mobile client device of claim 10, wherein the softwareprogram product-analyses messages from the messaging application andselects an indication sound automatically based on one or more ofcontents of the messages, senders of the messages, and recipients of themessages, using a text analyzer application.
 12. The mobile clientdevice of claim 10, wherein the software program product-provides aroute calculator that provides a combined route of driving and walkingbased on data from the email or calendar application comprising e-mails,calendar entries, a search application, and an Internet browser. 13-18.(canceled)
 19. The non-transitory computer readable memory of claim 1,wherein the messaging application is operated with machine learned datavia an artificial intelligence model.
 20. The non-transitory computerreadable memory of claim 4, wherein the messaging application isoperated with machine learned data via an artificial intelligence model.21. The mobile client device of claim 7, wherein the messagingapplication is operated with machine learned data via an artificialintelligence model.
 22. The mobile client device of claim 10, whereinthe messaging application is operated with machine learned data via anartificial intelligence model.